Project on Unsupervised Machine Learning:
Data Description
- The data contains features extracted from the silhouette of vehicles in different angles. Four "Corgie" model vehicles were used for the experiment: a double decker bus, Cheverolet van, Saab 9000 and an Opel Manta 400 cars. This particular combination of vehicles was chosen with the expectation that the bus, van and either one of the cars would be readily distinguishable, but it would be more difficult to distinguish between the cars.
- Since the target variable is related to distinguishing type of vehicle/class of vehicle like Car, Van and Bus, the, true positive here will be recognizing correctly and the accuracy of the models will be calculated based upon the recognition of the vehicle.
- The data at hand contains geometrical features or silhouette of 846 types of vehicles present on road. Out of which three clusters can be made based upon the class such as Car, Bus and Van.
- The purpose is to classify a given silhouette as one of three types of vehicle, using a set of features extracted from the silhouette. The vehicle may be viewed from one of many different angles.
</font>
Attribute Information
- All the features are geometric features extracted from the silhouette.
- All are numeric in nature.
 Compactness = (Avg. Perimeter) ^2 / Area
 Circularity = (Avg. Radius) ^2 / Area
 Distance Circularity = Area / (Avg. Distance from border) ^2
 Radius Ratio = (Max. Radius – Min. Radius) / Avg. Radius
 Pr. Axis Aspect ratio = (Minor Axis / Major Axis)
 Max. Length Aspect Ratio = (Length Perp Man Length) / (Man. Length)
 Scatter Ratio = (Inertia about Minor Axis) / (Inertia about Major Axis)
 Elongatedness = Area / (Shrink width) ^2
 Pr. Axis Rectangularity = Area / (Pr. Axis length * Pr. Axis Width)
 Max. Length Rectangularity = Area / (Max. length * Length perp. to this)
 Scaled Variance (Scaled variance along major axis) = (2nd order moment about major axis) / area
 Scaled Variance 1 (Scaled variance along minor axis) = (2nd order moment about minor axis) / area
 Scaled radius of Gyration = (Minor Variance + Minor Variance) / area
 Scaled radius of gyration.1: (3rd order moment about major axis) / (Sigma minor) ^3
 Skewness about: (3rd order moment about minor axis) / (Sigma Major) ^3
 Skewness about.1: (4th order moment about major axis) / (Sigma Minor) ^4
 Skewness about.2: (4th order moment about minor axis) / (Sigma Major) ^4
 Hollowness Ratio = (Area of Hollows) / (Area of bounding Polygon)
</font>
Objective
Main purpose is to classify the vehicles into the respective labels using the numerical attributes from the given geometric features extracted from Silhouette by applying dimensionlaity reduction technique - PCA and train a model using Principal components instead of training the model using just the raw data.
In [3]:
# importing the necessary package for performing advanced mathematical operation
import pandas as pd
# importing the necessary package for managing data
import seaborn as sns
import matplotlib.pyplot as plt
# importing the necessary packages for visualisation
sns.set (color_codes = True) # it will add a nice background to the graphs
%matplotlib inline
# commmand to tell Python to display my graphs
sns.set_style(style= 'darkgrid')
# pre-processing method
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
# methods and classes for evaluation
from sklearn.metrics import accuracy_score
from sklearn.metrics import recall_score
from sklearn import metrics
from sklearn.model_selection import cross_validate
import warnings
warnings.filterwarnings("ignore")
In [4]:
df = pd.read_csv('vehicle (1).csv')
In [5]:
df.head(20).T
Out[5]:
In [6]:
df.tail(20).T
Out[6]:
In [7]:
def indetailtable(df):
print(f'Dataset Shape: {df.shape}')
print('Total Number of rows in dataset= {}'.format(df.shape[0]))
print('Total Number of columns in dataset= {}'.format(df.shape[1]))
print('Various datatypes present in the dataset are: {}\n'.format(df.dtypes.value_counts()))
summary = pd.DataFrame(df.dtypes, columns = ['dtypes'])
summary = summary.reset_index()
summary['Name'] = summary['index']
summary['Name'] = summary[['Name', 'dtypes']]
summary['Misssing_values'] = df.isnull().sum().values
summary['Unique_values'] = df.nunique().values
summary['Duplicate_values'] = df.duplicated().sum()
summary['1st value'] = df.loc[0].values
summary['2nd Value'] = df.loc[1].values
summary['844th Value'] = df.loc[843].values
summary['845th Value'] = df.loc[844].values
return summary
In [8]:
brief = indetailtable(df)
brief
Out[8]:
Comments:¶
There are basically 845 number of data points / observations and 19 number of columns/features present in the dataset.
Out of which 14 number of rows are of float64 data type, four are integer (int64), and one is object. The column ‘Class’ is object datatype in nature and it contains three unique names in it such as van, bus and car.
There are lot of missing values present in the dataset, so we need to perform appropriate operation to avoid presence of missing values and their adverse effect on the model performance.
Similarly, there are no duplicate values present in the dataset, so it can also be treated as clean in terms of duplicate values.
Except the class column which is our target column, all other columns are filled with many number of unique values.
The status column which is object in terms of datatype has three unique values such as Van, Car and Bus.
In [9]:
# We will encode the Categorical Variable
from sklearn.preprocessing import LabelEncoder, OneHotEncoder
In [10]:
label_encoder = LabelEncoder()
df.iloc[:,18] = label_encoder.fit_transform(df.iloc[:,18]).astype('float64')
In [11]:
df.info()
In [12]:
df.columns
Out[12]:
In [13]:
df=df.rename(columns = {'pr.axis_aspect_ratio':'pr_axis_aspect_ratio', 'max.length_aspect_ratio':'max_length_aspect_ratio',
'pr.axis_rectangularity':'pr_axis_rectangularity', 'max.length_rectangularity':'max_length_rectangularity',
'scaled_variance.1':'scaled_variance_1', 'scaled_radius_of_gyration.1':'scaled_radius_of_gyration_1',
'skewness_about.1':'skewness_about_1', 'skewness_about.2':'skewness_about_2'})
df.columns
Out[13]:
In [14]:
for value in ['compactness', 'circularity', 'distance_circularity', 'radius_ratio',
'pr_axis_aspect_ratio', 'max_length_aspect_ratio', 'scatter_ratio',
'elongatedness', 'pr_axis_rectangularity', 'max_length_rectangularity',
'scaled_variance', 'scaled_variance_1', 'scaled_radius_of_gyration',
'scaled_radius_of_gyration_1', 'skewness_about', 'skewness_about_1',
'skewness_about_2', 'hollows_ratio', 'class']:
print(value,':', sum(df[value] == '?'))
- We don't have '?' symbol in the dataset
Dealing with missing values¶
In [15]:
# Any of the values in the dataframe is a missing Value?
df.isnull().values.any()
Out[15]:
In [16]:
# total number of missing values present in the entire dataframe..
df.isnull().sum().sum()
Out[16]:
In [17]:
# Replacing NaN with the mean of the column
df['circularity'].fillna(df.circularity.mean(),inplace = True)
In [18]:
# Replacing NaN with the mean of the column
df['distance_circularity'].fillna(df.distance_circularity.mean(),inplace = True)
In [19]:
# Replacing NaN with the mean of the column
df['radius_ratio'].fillna(df.radius_ratio.mean(),inplace = True)
In [20]:
# Replacing NaN with the mean of the column
df['pr_axis_aspect_ratio'].fillna(df.pr_axis_aspect_ratio.mean(),inplace = True)
In [21]:
# Replacing NaN with the mean of the column
df['scatter_ratio'].fillna(df.scatter_ratio.mean(),inplace = True)
In [22]:
# Replacing NaN with the mean of the column
df['pr_axis_rectangularity'].fillna(df.pr_axis_rectangularity.mean(),inplace = True)
In [23]:
# Replacing NaN with the mean of the column
df['scaled_variance'].fillna(df.scaled_variance.mean(),inplace = True)
In [24]:
# Replacing NaN with the mean of the column
df['scaled_variance_1'].fillna(df.scaled_variance_1.mean(),inplace = True)
In [25]:
# Replacing NaN with the mean of the column
df['scaled_radius_of_gyration'].fillna(df.scaled_radius_of_gyration.mean(),inplace = True)
In [26]:
# Replacing NaN with the mean of the column
df['scaled_radius_of_gyration_1'].fillna(df.scaled_radius_of_gyration_1.mean(),inplace = True)
In [27]:
# Replacing NaN with the mean of the column
df['skewness_about'].fillna(df.skewness_about.mean(),inplace = True)
In [28]:
# Replacing NaN with the mean of the column
df['skewness_about_1'].fillna(df.skewness_about_1.mean(),inplace = True)
In [29]:
# Replacing NaN with the mean of the column
df['skewness_about_2'].fillna(df.skewness_about_2.mean(),inplace = True)
In [30]:
# Replacing NaN with the mean of the column
df['elongatedness'].fillna(df.elongatedness.mean(),inplace = True)
In [31]:
# Any of the values in the dataframe is a missing Value?
df.isnull().values.any()
Out[31]:
In [32]:
# total number of missing values present in the entire dataframe..
df.isnull().sum()
Out[32]:
- Now the dataset is free from missing values.
- Hence, it is now ready for statistical analysis.
In [33]:
df.describe().T
Out[33]:
Comments:¶
In the above table 5 point summary have been described, i.e., minimum value, 25%, 50%, 75%, and maximum value.
• In the above statistical distribution analysis, it is noticed that, most of the features are having unique values and have a unique distribution.
• Each columns do have different units or scales. Some of the columns indicate measure of normal dimensions and some are ratios. Thus these columns have unique values and unique distribution as per their scales.
• The target column which is trinomial in nature has three categories or classes such as 0,1,2 for Bus, Car and Van respectively.
Further Checking individually
Univariate Analysis for Target Column¶
In [34]:
# Checking the value of counts for Status of PD
pd.value_counts(df['class'])
Out[34]:
Here
- 0.0 stands for Bus,
- 1.0 stands for Car, and
- 2.0 stands for Van
In [35]:
plt.figure(figsize = (20,15))
plt.subplot(2,2,1)
pd.value_counts(df['class']).plot(kind = 'bar', color = 'purple'); # to plot a bar chart
plt.xlabel('Types of Vehicles');
plt.subplot(2,2,2)
# Data to plot
labels = 'Car', 'Bus', 'Van'
sizes = [429, 218, 199]
colors = ['yellowgreen', 'violet', 'orange']
explode = (0.1, 0, 0.1) # explode 1st slice
# Plot
plt.pie(sizes,explode=explode, labels=labels, colors=colors, # to plot a pie chart
autopct='%1.1f%%',shadow=True, startangle= - 135)
plt.xlabel('Types of Vehicles');
plt.axis('equal')
plt.show()
In [36]:
df.groupby(['class']).count()
Out[36]:
Observations on Target Column:¶
Out of all the vehicle 50.7% are of Car, 23.5% are of van and 25.8% are of bus category.
The dataset is skewed in terms of target column, due to the uneven distribution of class of vehicles between car, bus and van. Car seems to be more on road than bas and van together.
And, here the main target or class of interest is to correctly identify the car, bus and van from their geometrical configuration.
Here, we do not want to misclassify any car as bus or van or vice versa, which may lead to bias the model during production. Thus our main aim is to minimize both type I and Type II error.
Univariate Analysis for Independent Columns like compactness, circularity, elongatedness, scaled_variance, scaled_radius_of_gyration, pr_axis_rectangularity, pr_axis_aspect_ratio, scatter_ratio¶
In [37]:
# Plotting the distribution of continous features individually
plt.figure(figsize = (20,15))
plt.subplot(3,3,1)
plt.hist(df.compactness, color = 'green', edgecolor = 'black', alpha = 0.7);
plt.xlabel('Compactness of Vehicle')
plt.subplot(3,3,2)
plt.hist(df.circularity, color = 'red', edgecolor = 'black', alpha = 0.7);
plt.xlabel('Measure of circularity of vehicle')
plt.subplot(3,3,3)
plt.hist(df.elongatedness, color = 'blue', edgecolor = 'black', alpha = 0.7);
plt.xlabel('Elongatedness of the Vehicle')
plt.subplot(3,3,4)
plt.hist(df.scaled_variance, color = 'orange', edgecolor = 'black', alpha = 0.7);
plt.xlabel('Variance of the vehicle after scaling along Major axis')
plt.subplot(3,3,5)
plt.hist(df.scaled_radius_of_gyration, color = 'purple', edgecolor = 'black', alpha = 0.7);
plt.xlabel('Radius of Gyration of vehicle')
plt.subplot(3,3,6)
plt.hist(df.pr_axis_rectangularity, color = 'violet', edgecolor = 'black', alpha = 0.7);
plt.xlabel('Measure of Rectangularity of the vehicle') ;
plt.subplot(3,3,7)
plt.hist(df.pr_axis_aspect_ratio, color = 'teal', edgecolor = 'purple', alpha = 0.7);
plt.xlabel('Aspect Ratio between Major Axis and Minor Axis') ;
plt.subplot(3,3,8)
plt.hist(df.scatter_ratio, color = 'brown', edgecolor = 'yellow', alpha = 0.7);
plt.xlabel('Scatter ratio: Ratio between inertia about Major & Minor Axis') ;
plt.subplot(3,3,9)
plt.hist(df.radius_ratio, color = 'white', edgecolor = 'red', alpha = 0.7);
plt.xlabel('Radius Ratio between Maximum and Minimum Radius');
Comments:
From above all histogram plots for various independent attributes it can be observed that none of them are normally distributed.
Though the aspect ratio column seems to be normally distributed, it is highly skewed towards right. Some of the attributes are multimodal in nature and it can be due to the mixing of Gaussian at different point of data collection.
In [38]:
# Plotting the distribution of continous features individually
plt.figure(figsize = (20,15))
plt.subplot(3,3,1)
sns.violinplot(df['compactness'], color = 'green')
plt.xlabel('Compactness of Vehicle')
plt.subplot(3,3,2)
sns.violinplot(df['circularity'], color = 'red')
plt.xlabel('Measure of circularity of vehicle')
plt.subplot(3,3,3)
sns.violinplot(df['elongatedness'], palette = 'gist_rainbow')
plt.xlabel('Elongatedness of the Vehicle')
plt.subplot(3,3,4)
sns.violinplot(df['scaled_variance'], color = 'orange')
plt.xlabel('Variance of the vehicle after scaling along Major axis')
plt.subplot(3,3,5)
sns.violinplot(df['scaled_radius_of_gyration'], color = 'purple')
plt.xlabel('Radius of Gyration of vehicle')
plt.subplot(3,3,6)
sns.violinplot(df['pr_axis_rectangularity'], color = 'violet')
plt.xlabel('Measure of Rectangularity of the vehicle') ;
plt.subplot(3,3,7)
sns.violinplot(df['pr_axis_aspect_ratio'], color = 'brown')
plt.xlabel('Aspect Ratio between Major Axis and Minor Axis') ;
plt.subplot(3,3,8)
sns.violinplot(df['scatter_ratio'], color = 'teal')
plt.xlabel('Scatter ratio: Ratio between inertia about Major & Minor Axis') ;
plt.subplot(3,3,9)
sns.violinplot(df['radius_ratio'], color = 'white')
plt.xlabel('Radius Ratio between Maximum and Minimum Radius');
From the above Violin plot, it can be inferred that, the columns 'Aspect ratio', 'scatter ratio' and 'radius ratio' are highly right skewed. 'Scaled variance' also seems to be right skewed.
A clear bimodal distribution curve can also be observed for the columns like 'Scaled Variance along Major axis', 'scatter ratio', 'Rectangularity measurement' and 'elongatedness' of the vehicle.
However, the order of modality among all these columns is not uniform. The aspect ratio attribute seems to be Normally distributed but it is again right skewed.
In [39]:
# Plotting the distribution of continous features individually
plt.figure(figsize = (20,15))
plt.subplot(3,3,1)
plt.hist(df.distance_circularity, color = 'green', edgecolor = 'black', alpha = 0.7);
plt.xlabel('Distance Circularity')
plt.subplot(3,3,2)
plt.hist(df.hollows_ratio, color = 'red', edgecolor = 'black', alpha = 0.7);
plt.xlabel('Hollowness Ratio')
plt.subplot(3,3,3)
plt.hist(df.max_length_aspect_ratio, color = 'blue', edgecolor = 'black', alpha = 0.7);
plt.xlabel('Maximum Length Aspect Ratio')
plt.subplot(3,3,4)
plt.hist(df.max_length_rectangularity, color = 'orange', edgecolor = 'black', alpha = 0.7);
plt.xlabel('Maximum Length Rectangularity')
plt.subplot(3,3,5)
plt.hist(df.scaled_variance_1, color = 'purple', edgecolor = 'black', alpha = 0.7);
plt.xlabel('Scaled Variance 1: Along Major Axis')
plt.subplot(3,3,6)
plt.hist(df.scaled_radius_of_gyration_1, color = 'violet', edgecolor = 'black', alpha = 0.7);
plt.xlabel('Scaled Radius of Gyration-1') ;
plt.subplot(3,3,7)
plt.hist(df.skewness_about, color = 'teal', edgecolor = 'brown', alpha = 0.7);
plt.xlabel('Skewness About') ;
plt.subplot(3,3,8)
plt.hist(df.skewness_about_1, color = 'brown', edgecolor = 'yellow', alpha = 0.7);
plt.xlabel('Skewness About 1') ;
plt.subplot(3,3,9)
plt.hist(df.skewness_about_2, color = 'white', edgecolor = 'red', alpha = 0.7);
plt.xlabel('Skewness About 2') ;
From the above histogram plot, inference can be drawn as follows: none of the columns are normally distributed. Though the column for 'maximum rectangularity' seems to be distributed normally, it is little bit right skewed.
Columns like 'Distance Circularity', 'Hollowness ratio', 'Maximum length rectangularity', 'skewness about-2' are having abundant amount of data points and can be expected to influence the target column.
Some of the columns like 'maximum length aspect ratio', 'scaled radius of gyratio-1' do not have much data points, and can be seen as right skewed. A discontinuity during the distribution period says the presence of outliers in those columns. it means these kind of columns can bias the target attribute and can also hamper the model performance in production.
In [40]:
# Plotting the distribution of continous features individually
plt.figure(figsize = (20,15))
plt.subplot(3,3,1)
sns.violinplot(df['distance_circularity'], color = 'green')
plt.xlabel('Distance Circularity')
plt.subplot(3,3,2)
sns.violinplot(df['hollows_ratio'], color = 'red')
plt.xlabel('Hollowness Ratio')
plt.subplot(3,3,3)
sns.violinplot(df['max_length_aspect_ratio'], palette = 'gist_rainbow')
plt.xlabel('Maximum Length Aspect Ratio')
plt.subplot(3,3,4)
sns.violinplot(df['max_length_rectangularity'], color = 'orange')
plt.xlabel('Maximum Length Rectangularity')
plt.subplot(3,3,5)
sns.violinplot(df['scaled_variance_1'], color = 'purple')
plt.xlabel('Scaled Variance 1: Along Major Axis')
plt.subplot(3,3,6)
sns.violinplot(df['scaled_radius_of_gyration_1'], color = 'violet')
plt.xlabel('Scaled Radius of Gyration-1') ;
plt.subplot(3,3,7)
sns.violinplot(df['skewness_about'], color = 'brown')
plt.xlabel('Skewness About') ;
plt.subplot(3,3,8)
sns.violinplot(df['skewness_about_1'], color = 'teal')
plt.xlabel('Skewness About 1') ;
plt.subplot(3,3,9)
sns.violinplot(df['skewness_about_2'], color = 'white')
plt.xlabel('Skewness About 2');
- From the above violin plot the inference drawn are as follows: Bimodal distribution can be observed in'Distance Circularity', 'Hollowness ratio' and 'scaled variance 1' along major axis, though the degree of modal is not same among all. The bimodal distribution curve shows the mixing of Gaussian, and also signifies a little amount of outliers.
- Distribution in the columns like 'Skewness about' and 'skewness about 1'can be treated as right skewed and also shows the presence of outliers. Distribution along 'Maximum length aspect ratio' and 'scaled radius of gyration1' seems to be weird, and can be due presence of outliers.
In [41]:
# for doing statistical calculation
import scipy
from sklearn import linear_model
import statsmodels.api as sm
from sklearn import metrics
from sklearn import datasets
import scipy.stats as stats
from scipy.stats import skew
# Preparing a pandas dataframe to store the skewness of each column.
Skewness = pd.DataFrame({'Skewness': [stats.skew(df.compactness), stats.skew(df.circularity),
stats.skew(df.distance_circularity),stats.skew(df.radius_ratio),
stats.skew(df.pr_axis_aspect_ratio), stats.skew(df.max_length_aspect_ratio),
stats.skew(df.scatter_ratio),stats.skew(df.elongatedness),
stats.skew(df.pr_axis_rectangularity),stats.skew(df.max_length_rectangularity),
stats.skew(df.scaled_variance),stats.skew(df.scaled_variance_1),
stats.skew(df.scaled_radius_of_gyration), stats.skew(df.scaled_radius_of_gyration_1),
stats.skew(df.skewness_about), stats.skew(df.skewness_about_1),
stats.skew(df.skewness_about_2), stats.skew(df.hollows_ratio)]},
index = ['compactness', 'circularity', 'distance_circularity', 'radius_ratio','pr_axis_aspect_ratio',
'max_length_aspect_ratio', 'scatter_ratio','elongatedness', 'pr_axis_rectangularity',
'max_length_rectangularity','scaled_variance', 'scaled_variance_1',
'scaled_radius_of_gyration','scaled_radius_of_gyration_1', 'skewness_about',
'skewness_about_1','skewness_about_2','hollows_ratio'])
Skewness
Out[41]:
- Columns like 'pr axis aspect ratio', 'max length aspect ratio' and 'scaled radius of gyration 1' are highly skewed towards right. This can be due to the presence of outliers in the independent columns. Out of all the columns, 'hollowness ratio' is only left skewed and this skewness can be overlooked during model building.
* Now to check the presence of outliers, box plot can be drawn for independent continous variables¶
In [42]:
plt.figure(figsize = (20,18))
plt.subplot(5,2,1)
sns.boxplot(x = df.compactness, color = 'green')
plt.xlabel('Compactness of Vehicle')
plt.subplot(5,2,2)
sns.boxplot(x = df.circularity, color = 'red')
plt.xlabel('Measure of circularity of vehicle')
plt.subplot(5,2,3)
sns.boxplot(x = df.elongatedness,palette = 'gist_rainbow')
plt.xlabel('Elongatedness of the Vehicle')
plt.subplot(5,2,4)
sns.boxplot(x = df.scaled_variance, color = 'orange')
plt.xlabel('Variance of the vehicle after scaling along Major Axis')
plt.subplot(5,2,5)
sns.boxplot(x = df.scaled_radius_of_gyration, color = 'purple')
plt.xlabel('Radius of Gyration of vehicle')
plt.subplot(5,2,6)
sns.boxplot(x = df.pr_axis_rectangularity, color = 'violet')
plt.xlabel('Measure of Rectangularity of the vehicle') ;
plt.subplot(5,2,7)
sns.boxplot(x = df.pr_axis_aspect_ratio, color = 'brown')
plt.xlabel('Aspect Ratio between Major Axis and Minor Axis') ;
plt.subplot(5,2,8)
sns.boxplot(x = df.scatter_ratio, color = 'teal')
plt.xlabel('Scatter ratio: Ratio between inertia about Major & Minor Axis') ;
plt.subplot(5,2,9)
sns.boxplot(x = df.radius_ratio, color = 'white')
plt.xlabel('Radius Ratio between Maximum and Minimum Radius');
- The inferences drawn from the above box plots as follows: Most of the columns are not affected by outliers except a few. As seen from the above histogram and violin plots the column 'pr axis aspect ratio' i.e. the aspect ratio between major axis and minor axis has been affected by outliers and a proper pre-processing can be imputed to the column in order to enhance the performance of the model. The columns like 'scaled variance' and 'radius ratio' are also been affected by outliers however the effect will not be that much sound on model accuracy.
- In addition, all the columns are not distributed uniformly, the median value of some of the columns are shifted towards right and for some of them median is situated towards left, it means for some of the columns 25% quantile has more data points and for some 75% quantile has more data points.
In [43]:
plt.figure(figsize = (20,18))
plt.subplot(5,2,1)
sns.boxplot(x = df.distance_circularity, color = 'green')
plt.xlabel('Distance Circularity')
plt.subplot(5,2,2)
sns.boxplot(x = df.hollows_ratio, color = 'red')
plt.xlabel('Hollowness Ratio')
plt.subplot(5,2,3)
sns.boxplot(x = df.max_length_aspect_ratio, palette = 'gist_rainbow')
plt.xlabel('Maximum Length Aspect Ratio')
plt.subplot(5,2,4)
sns.boxplot(x = df.max_length_rectangularity, color = 'orange')
plt.xlabel('Maximum Length Rectangularity')
plt.subplot(5,2,5)
sns.boxplot(x = df.scaled_variance_1, color = 'purple')
plt.xlabel('Scaled Variance 1: Along Major Axis')
plt.subplot(5,2,6)
sns.boxplot(x = df.scaled_radius_of_gyration_1, color = 'violet')
plt.xlabel('Scaled Radius of Gyration-1') ;
plt.subplot(5,2,7)
sns.boxplot(x = df.skewness_about, color = 'brown')
plt.xlabel('Skewness About') ;
plt.subplot(5,2,8)
sns.boxplot(x = df.skewness_about_1, color = 'teal')
plt.xlabel('Skewness About 1') ;
plt.subplot(5,2,9)
sns.boxplot(x = df.skewness_about_2, color = 'white')
plt.xlabel('Skewness About 2');
- Inferences drawn from the above box plots are as follows: Though some of the columns are affected by outliers, the columns like 'maximum length aspect ratio' and 'scaled radius of gyration 1' are highly affected by outliers and also a discontinuity can also be observed between outliers. This is a rare case in its own. The proper data pre-processing imputation is essentially required before using the dataset for model building.
- A few columns like 'Scaled variance 1: along major axis', 'skewness about' and 'skewness about 1' are having very less outliers, and these may not affect the performance of the model, however a proper pre-processing can give extra edge to the performance.
- Some of the columns are having median values towards right and some of them are having towards left. It means in some of the columns more data points are located within 25% quantile region or may be above 75% quantile region.
In [44]:
sns.pairplot(df, hue="class");
- In some of the columns, the data points are captured very close to each other in a cluttered manner and in some other columns a marginal space can be observed between data points. The presence of data points with some gap can influence the result of predicting different levels for target column.
- The presence of few data points can be seen in some columns like 'Max length aspect ratio', 'scaled radius of gyration 1' etc..
- Among all the 3 types’ vehicles, the bus class seems to be distributed normally or nearby to the normal with respect to all the independent variables.
- Some of the columns seem to be highly correlated, a linear correlation is observed between many of the columns.
- Distribution in Some of the columns are like scatter cloud with presence of outliers.
- A perfect collinearity is observed between many of the columns. Further, degree of correlation can be observed from correlation matrix or can be visualized from heat map made below for correlation.
- The distribution of different levels of target column is uneven in nature. Since the data points are more for the vehicle type car, the distribution of all the levels also dominated by car as well and the prediction accuracy of the model will also be biased towards the vehicle type car.
- Based upon the number of levels present in dataset, three number of clusters can be formed.
- Apart from the cluster made out of types of vehicles, based upon the modality of the distribution curve, we can also make two to four number of clusters.
In [45]:
df.corr().T
Out[45]:
In [46]:
colormap = plt.cm.viridis # Color range to be used in heatmap
plt.figure(figsize = (15,12))
plt.title('Pearson Correlation of attributes', y=1, size = 20)
sns.heatmap(df.corr(), linewidth = 0.2, vmax = 1.0,
square = True, cmap = colormap,linecolor = 'red', annot = True);
- None of the independent columns are highly correlated with the dependent column.
- Half of the columns present in the data set are highly correlated with each other. Some are correlated in a positive manner and some are correlated in a negative manner.
- But nowhere a perfect collinearity is observed among the columns. Thus none of the columns are duplicate in nature.
- A high collinearity among the independent columns signifies dependencies on each other. Means a simple change in value of any column directly or indirectly affect the corresponding correlated column.
- In order to avoid the dominant effect of these independent columns on model performance and to make it unbiased we can use level of significance and put a threshold limit on correlation value to drop or delete the columns having multicollinearity.
In [47]:
plt.figure(figsize = (20,24))
plt.subplot(4,3,1)
sns.scatterplot(df.compactness, df.circularity, hue = df['class'], palette = ['red', 'blue', 'orange'])
plt.title('compactness Vs. circularity', color = 'brown');
plt.subplot(4,3,2)
sns.scatterplot(df.radius_ratio, df.scatter_ratio, hue = df['class'], palette = ['purple', 'orange', 'green'])
plt.title('radius_ratio Vs. scatter_ratio', color = 'brown');
plt.subplot(4,3,3)
sns.scatterplot(df.elongatedness, df.compactness, hue = df['class'], palette = ['red', 'green', 'yellow']);
plt.title('elongatedness Vs. compactness', color = 'brown');
plt.subplot(4,3,4)
sns.scatterplot(df.elongatedness, df.circularity, hue = df['class'], palette = ['orange', 'green', 'violet']);
plt.title('elongatedness Vs. circularity', color = 'brown');
plt.subplot(4,3,5)
sns.scatterplot(df.scaled_radius_of_gyration_1, df.skewness_about_2, hue = df['class'], palette = ['blue', 'orange', 'brown']);
plt.title('scaled_radius_of_gyration_1 Vs. skewness_about_2 ', color = 'brown');
plt.subplot(4,3,6)
sns.scatterplot(df.scaled_radius_of_gyration_1, df.hollows_ratio, hue = df['class'], palette='nipy_spectral');
plt.title('scaled_radius_of_gyration_1 Vs.hollows_ratio', color = 'brown');
plt.subplot(4,3,7)
sns.scatterplot(df.skewness_about_2, df.hollows_ratio, hue = df['class'], palette='plasma_r');
plt.title('skewness_about_2 Vs. hollows_ratio', color = 'brown');
plt.subplot(4,3,8)
sns.scatterplot(df.pr_axis_rectangularity, df.max_length_rectangularity, hue = df['class'], palette = ['purple', 'green', 'orange']);
plt.title('pr_axis_rectangularity Vs. max_length_rectangularity', color = 'brown');
plt.subplot(4,3,9)
sns.scatterplot(df.pr_axis_aspect_ratio, df.scatter_ratio, hue = df['class'], palette = ['Blue', 'purple', 'green']);
plt.title('pr_axis_aspect_ratio Vs. scatter_ratio', color = 'brown');
plt.subplot(4,3,10)
sns.scatterplot(df.scaled_variance, df.scaled_variance_1, hue = df['class'], palette = ['violet', 'purple', 'green']);
plt.title('Scaled Variance along Major axis vs. Minor axis', color = 'brown');
plt.subplot(4,3,11)
sns.scatterplot(df.scaled_radius_of_gyration_1, df.skewness_about, hue = df['class'], palette = ['orange','purple', 'red']);
plt.title('scaled_radius_of_gyration_1 Vs. skewness_about', color = 'brown');
plt.subplot(4,3,12)
sns.scatterplot(df.skewness_about_1, df.skewness_about_2, hue = df['class'], palette = ['Blue', 'orange', 'green']);
plt.title('skewness_about_1 Vs. skewness_about_2', color = 'brown');
- The above scatter plots are drawn by taking some random independent columns in order to check the collinearity and cluster cloud.
- Except the scatter plot between 'Scaled variance 1' and 'scaled variance', none of the columns are having very high correlation value with each other.
- A scatter cloud can be observed in the very last plot, which signifies a very low correlation value almost near to zero. So this kind of correlation value can help to predict the accurate result without any bias error or variance error.
- In some of the scatter plots, presence of outliers can be observed, but the level seems to be different among all. Mostly, the vehicle type van seems to be outlier in all the above scatter plots.
Using Z-score method to standardize the data.¶
- The independent columns present in the dataset are numerical in nature. The range and scale are not uniform across the numerical columns. Thus to avoid any kind of biasness towards any of the numerical columns or to avoid the dominating effect of any column on the performance accuracy, we need to scale these before doing any operation.
- So, here we will be imputing Z-score method to scale the independent columns present in the dataset.
- Prior to that, we need no separate out our target column and independent columns.
In [48]:
df_ind_var = df.drop('class', axis =1) # Separating the target column from independent column
In [49]:
from scipy.stats import zscore
df_ind_var_z = df_ind_var.apply(zscore) # imputing zscore to the numerical columns.
In [50]:
df_ind_var_z.head()
Out[50]:
In [51]:
from sklearn.model_selection import train_test_split
X = df.drop('class', axis =1)
y = df['class']
In [52]:
X_z = X.apply(zscore)
X_z.head()
Out[52]:
In [53]:
x_train, x_test, y_train, y_test = train_test_split(X_z, y, test_size = 0.3, random_state = 533)
x_train.head()
Out[53]:
In [54]:
y_train.head()
Out[54]:
In [55]:
# checking the split of data
print('{0:0.2f}% data is in training set'.format((len(x_train)/len(df.index))*100))
print('{0:0.2f}% data is in testing set'.format((len(x_test)/len(df.index))*100))
SVM Classifier Model Building and prediction¶
In [56]:
# Building a Support Vector Machine on trian data
svc_model_rbf = SVC( kernel = 'rbf')
svc_fit_rbf = svc_model_rbf.fit(x_train, y_train)
y_train_pred_svc_rbf = svc_fit_rbf.predict(x_train)
print('Training model Accuracy value: {0:0.2f}%'.format(accuracy_score(y_train, y_train_pred_svc_rbf)*100))
y_test_pred_svc_rbf = svc_fit_rbf.predict(x_test)
print('Testing Model Accuracy value: {0:0.2f}%'.format(accuracy_score(y_test, y_test_pred_svc_rbf)*100))
SVM Classifier Confusion Matrix¶
In [57]:
cmSVC_rbf = metrics.confusion_matrix(y_test, y_test_pred_svc_rbf, labels = [1.0,2.0, 0.0])
df_cmSVC_rbf = pd.DataFrame(cmSVC_rbf, index = [i for i in ['Actual bus', 'Actual van', 'Actual car']],
columns = [i for i in ['Predict bus', 'Predict van', 'Predict car']])
colormap = plt.cm.viridis
plt.figure(figsize = (8,5))
sns.heatmap(df_cmSVC_rbf, annot = True, fmt = 'g', cmap = colormap, linecolor = 'red');
The confusion matrix Explanation:
- Total number of vehicles correctly predicted by the model are 128, 48 and 70 for bus, van and car respectively.
- Total number of vehicles incorrectly predicted by the model are 8 out of 846. This is the combination of both Type I error and Type II error.
SVM Classifier Model Report¶
In [58]:
print(metrics.classification_report (y_test, y_test_pred_svc_rbf, labels = [1.0,2.0,0.0]))
In [59]:
import numpy as np
y_grid = (np.column_stack([y_test, y_test_pred_svc_rbf])) # Checking the prediction value and actual value.
In [60]:
print(y_grid)
In [61]:
np.savetxt('ocr.txt', y_grid, fmt = '%s') # Saving the prediction value and actual value in a text file format.
In [62]:
resultsdf = pd.DataFrame({'Techniques Used': ['SVM with scaling'], 'Accuracy (%)': [accuracy_score(y_test, y_test_pred_svc_rbf).mean()*100], 'Std. Dev (%)': [accuracy_score(y_test, y_test_pred_svc_rbf).std()*100.0] })
resultsdf = resultsdf[['Techniques Used', 'Accuracy (%)', 'Std. Dev (%)']]
resultsdf
Out[62]:
- In this Support vector classifier we could able to achieve an accuracy score of 96.85% after scaling the data with the help of z-score.
- Further, we can also try to improve the performance of the algorithm after imputing various feature engineering methods.
SVM Classifier Model Building and prediction¶
In [63]:
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
num_folds = 25
seed = 235
kfold = KFold(n_splits = num_folds, random_state = seed)
svc_k_model = SVC ( kernel = 'rbf')
results = cross_val_score(svc_k_model, X, y, cv = kfold)
print(results)
print('\n')
print('Accuracy before applying z_score:%.3f%% (%.3f%%)'%(results.mean()*100.0, results.std()*100.0))
In [64]:
tempresultsdf = pd.DataFrame({'Techniques Used': ['SVM Kfold w/o scaling'], 'Accuracy (%)': [results.mean()*100], 'Std. Dev (%)': [results.std()*100.0] })
resultsdf = pd.concat([resultsdf, tempresultsdf])
resultsdf = resultsdf[['Techniques Used', 'Accuracy (%)', 'Std. Dev (%)']]
resultsdf
Out[64]:
- In this iteration we have imputed one of the non-exhaustive feature engineering process such as KFold Cross validation.
- After imputing the KFold CV we got an accuracy value of 68.919% with an standard deviation of 9.15%. It means our accuracy can range from 58% to 78%. Which is basically a range estimate for the performance measurement of the model.
- Here we have taken 25 number of folds, however, many other value for k can also be taken and tested..
- Further Improvement can also be made on the performance after scaling the data.
In [65]:
num_folds = 25
seed = 235
kfold = KFold(n_splits = num_folds, random_state = seed)
svc_k_model = SVC ( kernel = 'rbf')
results = cross_val_score(svc_k_model, X_z, y, cv = kfold)
print(results)
print('\n')
print('Accuracy after applying z_score:%.3f%% (%.3f%%)'%(results.mean()*100.0, results.std()*100.0))
In [66]:
tempresultsdf = pd.DataFrame({'Techniques Used': ['SVM Kfold with scaling'], 'Accuracy (%)': [results.mean()*100], 'Std. Dev (%)': [results.std()*100.0] })
resultsdf = pd.concat([resultsdf, tempresultsdf])
resultsdf = resultsdf[['Techniques Used', 'Accuracy (%)', 'Std. Dev (%)']]
resultsdf
Out[66]:
- In this iteration we have imputed one of the non-exhaustive feature engineering process such as KFold Cross validation.
- After imputing the KFold CV we got an accuracy value of 95.57% with an standard deviation of 2.45%. It means our accuracy can range from 94.14% to 98.99%. Which is basically a range estimate for the performance measurement of the model.
- Here we have taken 25 number of folds, however, many other value for k can also be taken and tested..
- Let's, consider only those columns, which are having correlation value more than 0.9
In [67]:
corr = df.corr()
corr
Out[67]:
In [68]:
columns =np.full((corr.shape[0],), True, dtype=bool)
for i in range(corr.shape[0]):
for j in range (i+1, corr.shape[0]):
if corr.iloc[i,j]>=0.90:
if columns[j] :
columns[j] = False
In [69]:
selected_columns = df.columns[columns]
selected_columns.shape
Out[69]:
In [70]:
selected_columns
Out[70]:
- Above columns are taken into consideritation after imputation of feature elimination based upon the correlation value more than 0.9
- Only 13 number of columns are selected based upon the feature elimination condition
In [71]:
df.columns
Out[71]:
- Above columns are original columns of the dataset
In [72]:
df1 = df[selected_columns]
# Assigning the selected columns after imputation of feature elimination
In [73]:
df_corr = df1.corr()
df_corr
# Checking for the correlation among thoese columns
Out[73]:
In [74]:
colormap = plt.cm.viridis # Color range to be used in heatmap
plt.figure(figsize = (15,12))
plt.title('Pearson Correlation of attributes', y=1, size = 20)
sns.heatmap(df1.corr(), linewidth = 0.2, vmax = 1.0,
square = True, cmap = colormap,linecolor = 'red', annot = True);
- Still we can observe a collinearity between columns, but this time it is less than 0.9, however we have not deleted the columns which are having a correlation value more than -0.9.
- Further we don't want to delete the columns, since there is some possibilities of missing valuable information from the dataset.
- Now the dataset has only those columns with correlation less than 0.9
Now selecting columns based on p-value¶
Here we will be selecting the columns based on how they affect the p-value. Here we are removing the column Class because it is the column we are trying to predict
In [75]:
selected_columns1 = selected_columns[:12].values
# removing the class column
In [76]:
selected_columns1
Out[76]:
- Here we will be imputing the OLS (Ordinary Least Square) method to do the feature elimination based upon the Significant Confidence level of 0.05.
- Here our Assumption for Null Hypothesis will be 'The selected combination of dependent variable do not have any effect on the independent variable'.
- Here we will be building a small regression model and calculate the p values.
- If the p values are higher than the threshold limit (Here we have taken 0.05), we discard that combination of features.
In [77]:
import statsmodels.api as sm
def backwardElimination(x, y, sl, columns):
numvars = len(x[0])
for i in range(0, numvars):
regressor_OLS = sm.OLS(y,x).fit()
maxvar = max(regressor_OLS.pvalues).astype(float)
if maxvar > sl:
for j in range (0, numvars - i):
if (regressor_OLS.pvalues[j].astype(float) == maxvar):
x = np.delete(x,j,1)
columns = np.delete(columns,j)
regressor_OLS.summary()
return x, columns
In [78]:
SL = 0.05 # Threshold limit is 0.05
data_modeled, selected_columns2 = backwardElimination(df1.iloc[:,:12].values, df1.iloc[:,12].values, SL, selected_columns1)
In [79]:
# Moving the result to a new dataframe
result = pd.DataFrame()
result['class'] = df.iloc[:,18]
In [80]:
result.head(10).T
Out[80]:
In [81]:
# Creating a dataframe with the columns selected using the p-value and correlation
df2 = pd.DataFrame(data = data_modeled, columns = selected_columns2)
df2.head()
Out[81]:
Visualizing the selected features¶
Plotting the data to visualize their distribution
In [82]:
# Creating a dataframe with the columns selected using correlation alone
data_modeled1 = df1.iloc[:, :12].values
df3 = pd.DataFrame(data = data_modeled1, columns = selected_columns1)
df3.head()
Out[82]:
In [83]:
fig = plt.figure(figsize = (20,25))
j = 0
for i in df3.columns:
plt.subplot(6,4, j+1)
j+= 1
sns.distplot(df3[i][result['class']==0], color = 'r', label = 'Bus')
sns.distplot(df3[i][result['class']==1], color = 'b', label = 'Car')
sns.distplot(df3[i][result['class']==2], color = 'g', label = 'Van')
plt.legend(loc = 'best')
fig.suptitle('Vechile Data Analysis after feature elimination with using only correlation')
fig.tight_layout()
fig.subplots_adjust(top=0.95)
plt.show()
- The above group of graphs contain the features after completing feature elimination with the help of correlation among them.
- Here, we have taken a threshold value for correlation as 0.9, and eliminated the one of those columns which has a correlation value of 0.9.
- Here, we have also missed lot of information while deleting those columns.
In [84]:
fig = plt.figure(figsize = (20,25))
j = 0
for i in df2.columns:
plt.subplot(6,4, j+1)
j+= 1
sns.distplot(df2[i][result['class']==0], color = 'r', label = 'Bus')
sns.distplot(df2[i][result['class']==1], color = 'b', label = 'Car')
sns.distplot(df2[i][result['class']==2], color = 'g', label = 'Van')
plt.legend(loc = 'best')
fig.suptitle('Vechile Data Analysis after feature elimination with both p value and correlation')
fig.tight_layout()
fig.subplots_adjust(top=0.95)
plt.show()
- The above group of graphs contain the features after completing feature elimination with the help of correlation value among them and p-value of the individual.
Here, we have taken a threshold value for correlation as 0.9, and eliminated the one of those columns which has a correlation value of 0.9, and a p-value of 0.05.
Here, we have also missed lot of information while deleting those columns.
- Now we will split the data into train and test set. 30% of the data is used to create the test data and 70% to create train data.
In [85]:
x_train2, x_test2, y_train2, y_test2 = train_test_split(df2.values, result.values, test_size = 0.3, random_state = 256)
Building a model with the selected features¶
We are using a Support Vector Classifier with a RBF kernel to make the predictions. we will train the model on our training data and calculate the accuracy on test data.
SVM Classifier Model Building and prediction¶
In [86]:
# Building a Support Vector Machine on trian data
svc_model_rbf2 = SVC( kernel = 'rbf')
svc_fit_rbf2 = svc_model_rbf2.fit(x_train2, y_train2)
y_train_pred_svc_rbf2 = svc_fit_rbf2.predict(x_train2)
print('Training model Accuracy value: {0:0.2f}%'.format(accuracy_score(y_train2, y_train_pred_svc_rbf2)*100))
y_test_pred_svc_rbf2 = svc_fit_rbf2.predict(x_test2)
print('Testing Model Accuracy value: {0:0.2f}%'.format(accuracy_score(y_test2, y_test_pred_svc_rbf2)*100))
SVM Classifier Confusion Matrix¶
In [87]:
from sklearn.metrics import confusion_matrix
cmSVC_rbf2 = metrics.confusion_matrix(y_test2, y_test_pred_svc_rbf2, labels = [0,1,2])
df_cmSVC_rbf2 = pd.DataFrame(cmSVC_rbf2, index = [i for i in ['Actual bus', 'Actual car', 'Actual van']],
columns = [i for i in ['Predict bus', 'Predict car', 'Predict van']])
colormap = plt.cm.viridis
plt.figure(figsize = (8,5))
sns.heatmap(df_cmSVC_rbf2, annot = True, fmt = 'g', cmap = colormap, linecolor = 'red');
The confusion matrix Explanation:
- Total number of vehicles correctly predicted by the model are 0, 40 and 103 for bus, van and car respectively.
- Total number of vehicles incorrectly predicted by the model are 111 out of 846. This is the combination of both Type I error and Type II error.
SVM Classifier Model Report¶
In [88]:
print(metrics.classification_report (y_test2, y_test_pred_svc_rbf2, labels = [0,1,2]))
In [89]:
tempresultsdf = pd.DataFrame({'Techniques Used': ['SVM after feature elimination w/o scaling'], 'Accuracy (%)': [accuracy_score(y_test2, y_test_pred_svc_rbf2).mean()*100], 'Std. Dev (%)': [accuracy_score(y_test2, y_test_pred_svc_rbf2).std()*100.0] })
resultsdf = pd.concat([resultsdf, tempresultsdf])
resultsdf = resultsdf[['Techniques Used', 'Accuracy (%)', 'Std. Dev (%)']]
resultsdf
Out[89]:
- Here we got very less accuracy because of the less number of columns we have after feature elimination.
- Here we have also not imputed the z-score to scale the data, it means the result is highly biased and dominated by the columns with high scale.
- To scale the dataset, we will be imputing the z-score and again check for accuracy score
In [90]:
df2_z = df2.apply(zscore)
df2_z.head(3)
Out[90]:
In [91]:
x_train2_z, x_test2_z, y_train2_z, y_test2_z = train_test_split(df2_z.values, result.values, test_size = 0.3, random_state = 256)
SVM Classifier Model Building and prediction¶
In [92]:
# Building a Support Vector Machine on trian data
svc_model_rbf2_z = SVC( kernel = 'rbf')
svc_fit_rbf2_z = svc_model_rbf2_z.fit(x_train2_z, y_train2_z)
y_train_pred_svc_rbf2_z = svc_fit_rbf2_z.predict(x_train2_z)
print('Training model Accuracy value: {0:0.2f}%'.format(accuracy_score(y_train2_z, y_train_pred_svc_rbf2_z)*100))
y_test_pred_svc_rbf2_z = svc_fit_rbf2_z.predict(x_test2_z)
print('Testing Model Accuracy value: {0:0.2f}%'.format(accuracy_score(y_test2_z, y_test_pred_svc_rbf2_z)*100))
SVM Classifier Confusion Matrix¶
In [93]:
from sklearn.metrics import confusion_matrix
cmSVC_rbf2_z = metrics.confusion_matrix(y_test2_z, y_test_pred_svc_rbf2_z, labels = [0,1,2])
df_cmSVC_rbf2_z = pd.DataFrame(cmSVC_rbf2_z, index = [i for i in ['Actual bus', 'Actual car', 'Actual van']],
columns = [i for i in ['Predict bus', 'Predict car', 'Predict van']])
colormap = plt.cm.viridis
plt.figure(figsize = (8,5))
sns.heatmap(df_cmSVC_rbf2_z, annot = True, fmt = 'g', cmap = colormap, linecolor = 'red');
The confusion matrix Explanation:
Total number of vehicles correctly predicted by the model are 66, 60 and 117 for bus, van and car respectively.
Total number of vehicles incorrectly predicted by the model are 11 out of 846. This is the combination of both Type I error and Type II error.
This is done after feature elimination.
SVM Classifier Model Report¶
In [94]:
print(metrics.classification_report (y_test2_z, y_test_pred_svc_rbf2_z, labels = [0,1,2]))
In [95]:
tempresultsdf = pd.DataFrame({'Techniques Used': ['SVM after feature elimination with scaling'], 'Accuracy (%)': [accuracy_score(y_test2_z, y_test_pred_svc_rbf2_z).mean()*100], 'Std. Dev (%)': [accuracy_score(y_test2_z, y_test_pred_svc_rbf2_z).std()*100.0] })
resultsdf = pd.concat([resultsdf, tempresultsdf])
resultsdf = resultsdf[['Techniques Used', 'Accuracy (%)', 'Std. Dev (%)']]
resultsdf
Out[95]:
- The above SVC model has predicted the accuracy with only 10 number of columns out of 18 numbers.
- It gave an accuracy of 56.299% without scaling and 95.66% with scaling.
- So, this SVC model after feature elimination process seems to be much more reliable than the previous SVC model which has used all the independent columns to predict the accuracy.
In [96]:
x_train_z, x_test_z, y_train, y_test = train_test_split(X_z, y, test_size = 0.3, random_state = 533)
x_train_z.head()
Out[96]:
Step 1:¶
- Generating the covariance matrix and the eigen values for the PCA analysis
In [97]:
cov_matrix = np.cov(X_z.T) # the relevent covariance matrix
print('Covariance Matrix \n%s', cov_matrix)
Step 2:¶
- Generating the Eigen Values and Eigen Vectors
In [98]:
e_vals, e_vecs = np.linalg.eig(cov_matrix)
print('Eigenvectors \n%s'%e_vecs)
print('\nEigenValues \n%s'%e_vals)
Step 3:¶
- Finding out variance and cumulative variance by each eigen vectors
In [99]:
tot = sum(e_vals)
var_exp = [(i/ tot)* 100 for i in sorted(e_vals, reverse = True)]
cum_var_exp = np.cumsum(var_exp)
print('Cumulative Variance Expalained', cum_var_exp)
In [123]:
plt.plot(var_exp, color = 'red');
- Visually we can observe that their is a steep drop in variance explained with increase in number of PC's.
- We will proceed with 9 components her. But depending upon requirement of 95% variation or 7 components will also be good. m
In [121]:
# Scree Plot
plt.figure(figsize = (10,5))
plt.bar(range(1, e_vals.size +1), var_exp, alpha = 0.5, align = 'center', label = 'Individaul explained variance', color = 'red')
plt.step(range(1, e_vals.size+1), cum_var_exp, where = 'mid', label = 'Cumulative explained Variance', color = 'green')
plt.ylabel('Expalind Variance Ratio')
plt.xlabel('Principal Components')
plt.legend(loc = 'best')
plt.tight_layout()
plt.show()
Use PCA command from sklearn and find Principal Components. Transform data to components formed¶
In [102]:
# Importing scikit learn PCA.
# It does all the above steps and maps data to PCA dimensions in one shot
from sklearn.decomposition import PCA
# Note: We are generating only 9 PCA dimenisons (Dimensionalaity reduction from 18 to 9)
pca = PCA(n_components = 9)
X_z_PCA = pca.fit_transform(X_z)
X_z_PCA.transpose()
Out[102]:
In [103]:
pca.components_
Out[103]:
Finding out correlation between components and features¶
In [104]:
sns.pairplot(pd.DataFrame(X_z_PCA));
- A cluster cloud can be seen between all the independent Principal Components, which indicates a generic relationship among the independent variables.
- Now they have very less collinearity value among them, and they won’t affect the prediction of the target column much.
In [105]:
df_comp = pd.DataFrame(pca.components_, columns = list(X_z))
df_comp.head(3)
Out[105]:
In [106]:
colormap = plt.cm.viridis # Color range to be used in heatmap
plt.figure(figsize = (15,12))
plt.title('Pearson Correlation of attributes', y=1, size = 20)
sns.heatmap(df_comp.corr(), linewidth = 0.2, vmax = 1.0,
square = True, cmap = colormap,linecolor = 'red', annot = True);
- The above heat map correlation graph shows the correlation between columns after imputation of PCA.
- A decrease in correlation value can be observed among columns.
In [107]:
plt.figure(figsize = (12,6))
sns.heatmap(df_comp, cmap = 'viridis');
- The above heatmap shows the correlation between Principal Components and the original variables.
- Except a few boxes, all look like greenish or bluish. It means, a very little correlation can be observed between each other,
- PC no. 4 seems to be more correleated to skewness_about_1 column in a positive manner.
- PC. no. 1 seems to be more correlated to skewnwss_about_2 and hollows_ratio columns in a negative manner.
In [108]:
x_train_pca, x_test_pca, y_train_pca, y_test_pca = train_test_split(pd.DataFrame(X_z_PCA), y, test_size = 0.3, random_state = 533)
y_train_pca.head()
Out[108]:
In [109]:
x_test_pca.head(3)
Out[109]:
In [110]:
# checking the split of data
print('{0:0.2f}% data is in PCA training set'.format((len(x_train_pca)/len(df.index))*100))
print('{0:0.2f}% data is in PCA testing set'.format((len(x_test_pca)/len(df.index))*100))
SVM Classifier Model Building and prediction¶
In [111]:
# Building a Support Vector Machine on trian data
svc_model_rbf_pca = SVC( kernel = 'rbf')
svc_fit_rbf_pca = svc_model_rbf_pca.fit(x_train_pca, y_train_pca)
y_train_pred_svc_rbf_pca = svc_fit_rbf_pca.predict(x_train_pca)
print('Training model Accuracy value after PCA: {0:0.2f}%'.format(accuracy_score(y_train_pca, y_train_pred_svc_rbf_pca)*100.0))
y_test_pred_svc_rbf_pca = svc_fit_rbf_pca.predict(x_test_pca)
print('Testing Model Accuracy value after PCA: {0:0.2f}%'.format(accuracy_score(y_test_pca, y_test_pred_svc_rbf_pca)*100.0))
SVM Classifier Confusion Matrix¶
In [112]:
cmSVC_rbf_pca = metrics.confusion_matrix(y_test_pca, y_test_pred_svc_rbf_pca, labels = [1.0,2.0,0.0])
df_cmSVC_rbf_pca = pd.DataFrame(cmSVC_rbf_pca, index = [i for i in ['Actual car', 'Actual van', 'Actual bus']],
columns = [i for i in ['Predict car', 'Predict van', 'Predict bus']])
colormap = plt.cm.viridis
plt.figure(figsize = (8,5))
sns.heatmap(df_cmSVC_rbf_pca, annot = True, fmt = 'g', cmap = colormap, linecolor = 'red');
SVM Classifier Model Report¶
In [113]:
print(metrics.classification_report (y_test_pca, y_test_pred_svc_rbf_pca, labels = [1.0,2.0,0.0]))
In [114]:
tempresultsdf = pd.DataFrame({'Techniques Used': ['SVM after PCA imputation'], 'Accuracy (%)': [accuracy_score(y_test_pca, y_test_pred_svc_rbf_pca).mean()*100], 'Std. Dev (%)': [accuracy_score(y_test_pca, y_test_pred_svc_rbf_pca).std()*100.0] })
resultsdf = pd.concat([resultsdf, tempresultsdf])
resultsdf = resultsdf[['Techniques Used', 'Accuracy (%)', 'Std. Dev (%)']]
resultsdf
Out[114]:
- After imputation of PCA, the accuracy is observed to be 95.27%
- This is far better than all others, because it has only taken 8 components to give an accuracy of 95% which is comparatively very less than all other methods we have used to calculate the accuracy.
- Hence, this seems to be more feasible.
- we will also further try to enhance the accuracy by using KFold CV.
SVM Classifier¶
In [115]:
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
num_folds = 25
seed = 235
kfold_pca = KFold(n_splits = num_folds, random_state = seed)
svc_k_model_pca = SVC ( kernel = 'rbf')
results_pca = cross_val_score(svc_k_model, X_z_PCA, y, cv = kfold)
print(results_pca)
print('\n')
print('Accuracy after performing PCA:%.3f%% (%.3f%%)'%(results_pca.mean()*100.0, results_pca.std()*100.0))
In [116]:
tempresultsdf = pd.DataFrame({'Techniques Used': ['SVM after PCA imputation with K-fold'], 'Accuracy (%)': [results_pca.mean()*100], 'Std. Dev (%)': [results_pca.std()*100.0] })
resultsdf = pd.concat([resultsdf, tempresultsdf])
resultsdf = resultsdf[['Techniques Used', 'Accuracy (%)', 'Std. Dev (%)']]
resultsdf
Out[116]:
- After imputation of PCA with KFold CV, the accuracy is observed to be 95.629%, with a standard deviation of 2.77%
- It means the accuracy can range between 92.99% and 98.3% in real.
- This is far better than all others, because it has only taken 8 components to give an accuracy of 95.629% which is comparatively very sound than all other methods we have used to calculate the accuracy.
- Hence, this seems to be more feasible.
In [117]:
resultsdf
Out[117]:
Comparing Overall Accuracy of various models graphically......¶
In [118]:
fig = plt.figure(figsize = (22,5))
plt.title ('Accuracy values for various models/techniques',y=1, size = 22, color = 'red')
sns.barplot(y = resultsdf['Accuracy (%)'], x = resultsdf['Techniques Used'], facecolor = (0.5,0.5,0.5,0.8), linewidth = 10, edgecolor = sns.color_palette ('dark', 12) );
plt.ylabel('Accuracy in %', size = 20)
plt.xlabel('Methodologies applied to calculate performance measurement', size = 20)
plt.tight_layout()
Here we have compared the accuracy scores of SVC models built with raw data and Principal Components.
The comparison is also drawn among the SVC models built with K-Fold cross validation feature engineering technique.
- The inferences are as follows:
The main point to be considered here is the difference in independent attributes for raw data and Principal components. In raw data we have total 18 number of independent attributes, whereas in case of PCA, we collected total 7 number of independent components which captured about 95% of the variance explained.
Thus we moved with a little more, by taking an extra attribute, and built the model with total 8 number of PCs or independent columns and subsequently calculated the accuracy.
The model built with raw data gave an accuracy score of 96.85% and the model built with Principal Components gave an accuracy of 95.27%.
Similarly we have used K-Fold CV to calculate the accuracy of the models by taking both raw data and Principal Components.
Hence, we got an accuracy of 68.919% while building the model with raw data without scaling, an accuracy of 96.57% while building the model with raw data after scaling, and got an accuracy of 95.629% while building the model with Principal Components.
For K-Fold CV, we have taken total 25 number of folds i.e. the value of K as 25.
However, any number of value can be taken based upon the number of data points, independent features and our convenience.
We have also reflected the range estimate values for K-Fold cross validation technique, however, the same was not possible with train test split method so we imitated the point estimate value of accuracy.
Here, we have also tried to do some feature engineering activities by imputing the process like OLS (Ordinary Least Square) and feature elimination based upon the Correlation value among the columns and p-value of independent columns.
Thus after imputing the feature elimination process, we eliminated the columns and end up with only 10 columns. So after building the model and testing we achieved the accuracy score of around 56.29% before scaling the dataset and 95.66% after scaling the dataset.
Thus at the end we can reach out to the conclusion that, the SVC model with PCA and K-Fold has the best accuracy score among all the models. This also gives a range estimate of the accuracy at 95% confidence level.
It means the accuracy can vary between the standard deviation of 2.77% from both negative and positive sides.
Since, we have an accuracy score of 95.629 % with corresponding standard deviation value of 2.77% we can give a range estimate of accuracy between 92.505 % and 98.399 %. And this is with a confidence level of 95 %. That means with 95 % of confidence level we can say our model accuracy will be between 92.505 % and 98.399 % in production.
The cause of considering the above explained model is less number of columns. i.e. with only 8 number of columns it is giving an average accuracy of 95.629 %, and due to the presence of less number of attributes the model is also less complex and least prone to overfitting. Thus, with a very simple model we are achieving the best accuracy score. Hence the SVC model with PCA and K-Fold CV is the best model for production.